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ABSTRACT 


Developed along with a tape-slide package, the guide 



covers evaluation of special educational programs. Robert McIntyre 
discusses evaluation for decision making; Victor Baldwin considers 
sources of help and how to use them; and Helmut Hofmann treats 
objectives as guidelines for action, data collection, and budget 
planning and evaluation. Wesley Meierhenry describes what and when to 
evaluate as well as reporting and dissemination; he also lists 
references on evaluation*. (JD) 
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INTRODUCTION 



Evaluation of educational programs has become one of the most 
critical concerns in all of education today. In spite of the efforts 
and resources which are being applied to produce new and innovative 
educational programs and activities, it is almost impossible to 
determine whether one is more effective, acceptable, and/or efficient 
than another because of the lack of adequate evaluation procedures. 

Nowhere is the dilemna greater than in the field of special 
education. Although a great variety of programs and activities have 
been designed to improve learning opportunities for students in special 
education programs varying from the addition of an 8 mm film in a 
traditional classroom setting to practically all elements of instruc- 
tion being different including new facilities especially designed to 
deal with the specific handicap, a teaching staff completely retrained 
as a team and instructional materials especially prepared for the 
program and the group of youngsters to be served. The above two 
examples suggest a range of instructional innovations from the simple 
addition of one variable to a multiplicity of variables each interacting 
with the other. Unfortunately, in neither case have we been in a position 
to report much useful information about either approach which would 
assist in determining if the innovation should be continued in the 
present location or whether it should be attempted in new situations. 



Thu purpose cf this Resource Guide is to provide assistance to 
teachers, project directors, administrators, state directors, and in 
fact, anyone who has the responsibility of making decisions about 
whether to initiate, promote, introduce, and/or judge a program for 
handicapped children. The Resource Guide attempts to dispel the 
notion that evaluation is something which is so complex and esoteric 
so that only researchers and evaluation specialists are capable of 
conducting it. Evaluation may or may not always include all the 
elements of research but even where it does, Charles F. Kettering 
once said that "Research is a high-hit word that scares a lot of 
people. It needn’t.. .research is nothing but a state of mind — a 
friendly, welcoming attitude toward change.” 

It is the hope that those who use this Resource Guide will 
"change their minds" both in their view of what evaluation is all 
about as well as how to plan for and carry out evaluation from the 
simplest to the most complex educational innovations. Whether the 
authors were successful will be determined by those of you who use 
this Guide , We solicit your feedback about strengths and weaknesses, 
where it communicates and where it doesn’t and where deletions or 
additions should be made. A major theme of the Resource Guide is 
that evaluation takes place in order that decision-making may be 
improved. Your comments will assist in decision-making about what 
future, if any, there should be to this initial effort. 

Fortunately, the writers of this Guide have received already 
much more feedback than is usually the case with similar publications. 

The first formal evaluation of it occurred at San Francisco on May 13, 1969, 



when a selected group of specialists from all types of programs and 
activities in special education reacted to a preliminary edition. 

Their comments led to both organizational as well as substantive 
changes „ A second formal evaluation took place in connection with the 
State Directors of Special Education in a meeting on July 11, 1969, at 
Jackson, Mississippi. This evaluation, like the first, produced many 
helpful suggestions. In addition to these two formal evaluations, 
many other knowledgeable persons have submitted useful suggestions. 

As Project Director, I wish to recognize the contributions of 
each member of the writing team. All were dedicated to the proposition 
that students, instruction, and education in general would all be 
served better if evaluation procedures could be improved. The fact 
that this Resource Guide was brought from an idea, through conceptualiza- 
tion, writing, field testing, and revision to this first edition in 
less than six months is testimony to the commitment and expertness of 
each member of the writing team. 

The Project Director wishes to acknowledge the very great 
assistance and encouragement of a number of staff members of the 
Education for the Handicapped Branch of the U. S. Office of Education, 
includ ing especially Tom Irvin, Chester Avery, and Frank Withrow. 



Wesley C. Meierhenry, Project Director 
University of Nebraska 
July 30, 1969 



Chapter I 

EVALUATION FOR DECISION WAKING 

If you 4on ’t know where you’ve going, you’ve likely tc end up somewhere else . 

As educators, we are finding that we have to evaluate and report 
on our programs in many different ways to different groups. We have 
always evaluated what we were doing, sometimes informally and without 
systematic plan. We have always reported to parents, principals, and 
other teachers. Now we find that boards of education; state legislatures, 
and agencies of the federal government are asking for reports on the 
effectiveness of our programs, particularly those which they fund. 

Although our basic process remains the same, we need to be aware of the 
factors which determine techniques of collecting and reporting infor- 
mation : 

1) Determining what and when to evaluate 

2) Developing objectives as guidelines for action 

3) Collecting data 

4) Relating budget plans to evaluation 

5) Using consultants 

6) Reporting and disseminating results 

The purpose of this guide is to provide some directions for evaluating 
our programs in special education. 

Evaluation is a word with many connotations (and at least a couple 
of meanings). Dictionaries define the verb "to evaluate" as determining 
worth. This meaning suggests to educators the scoring of tests, giving 
of grades, figuring percentages and various other exercises in arithmetic. 



A second Meaning includes the judgmental activity of placing a value. 
Va.luing, as a part of evaluation, is not necessarily empirical, but 
rests on personal preferences and orientations. Both evaluating and 
valuing are involved in any attempt to see how we are doing in trying 
to educate children. 

We may look at a particular program for a group of students and 
find that 65% of the kids are more than two years below national 
expectancies, for example. We may then decide that this is terrible 
and that we must do something about it. 

Evaluation and the Teaching-Learning Process 

Evaluation comes at the start of an educational process. When 
we meet with a new class we spend some time finding out who and where 
the children are. We may give tests, either standardized or our own. 

We may ask the pupils to do things — read, write, or recite. We then 
form an opinion about where to begin teaching. As we teach, we continue 
to collect information about the pupils’ progress. If all goes well 
we may step up the pace or add extra aspects. If progress is slow we 
may go back and review looking for other ways to present the material. 

When we finish a unit we again evaluate in order to provide data about 
the child and about our teaching. 

This process of observing and gathering data about students ’ behavior 
and progress toward specified objectives is used to formulate a course 
of action, whether we are teachers, state directors of special education, 
or U. S. 0. E. officials. 

Evaluation and Decision Making 



In all cases the purpose of evaluation is to get data for making 
decisions. Too many educational decisions in the past have been based on 



tradition or expert opinion. Many of us are involved in projects 
which, by definition, are innovative. We have no past experience 
upon which to lean. We need to make decisions! decisions about 
where to put children, how to distribute our energies and resources, 
in the priority of order, and whether the things we are doing are 
helping us reach the goals we have chosen. Our evaluation process 
ideally should be giving us the data needed for these decisions. Also 
we should be collecting only the kinds of data useful in decision 
making. The arduous task of collecting data which are not immediately 
useful to us has sometimes given evaluation a bad name and has caused 
some educators to feel that data collection is divorced and separate from 
the real concerns of the schools. 

Evaluation and Benefit to Children 

The ultimate test of a program’s effectiveness is the extent to 
which it has changed the behavior and capabilities of a child. Where- 
ever possible, data concerning changes should be used for evaluation. 

The evaluation procedure should match the program’s objectives- and 
where the program is designed to make specific changes in youngsters, 
these changes should be the basis for evaluation. For example, if we 
have established a program to increase the verbal production of children, 
we might evaluate it with an external test such as an Illinois Psycho- 
linguistic sub— test or with an internal measure , such as recording the 
frequency and length of the child’s contributions in class discussion. 
Where the project is designed to change some other element in the 

Alt should be noted that this is a requirement under Title VI-A 
and the PL 89-313 amendment to Title I of the ESEA. Copies of section 
F on program and project design and evaluation from the Office of 
Education administrative manuals for these two programs are available 
from the Division of Services, Bureau of Education for the Handicapped, 
U. S. Office of Education, Washington, D. C. 20202. 



instructional system — teachers, availability of aides, use of 
instructional technology, administrative procedures — the evaluation 
should focus on that particular element directly, and indirectly on 
the changes in pupils, if any. For example, if we have approached the 
goal of changing behavior and capabilities in children through pro- 
viding in-service education to our teachers, we may wish to evaluate 
the effectiveness of this training before it is possible to observe 
changes in children. In this case teachers' increased knowledge can 
be evaluated. Have they increased verbalization in class? Look at 
their first week’s plans. Are they allowing increased time for group 
discussion? Are room arrangements different from before and more 
likely to support interaction between students? 

Children are the major, but not the only, consumers of our 
educational "product Other elements also have a stake in education 
and can also provide legitimate data for evaluation. Programs may 
show their effectiveness in many ways: greater parent satisfaction in 

the changes in a school’s pattern of special education referrals, in 
the development of special classes moving in closer interaction with 
other students, or even in greater community support. 

Evaluation and Research 

Evaluation and research can be considered a continuum with "pure" 
research near one end and the daily informal evaluation of on-going 
act.’vitirs on che other. Most research and evaluation in education 
fa 1 Is somewhere in the middle. Realizing this, let’s compare the two 
extremes to better understand the range of processes. 
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First, pure research, and most applied research aims at results 
which can be applied to samples in another larger population. The 
researcher may collect data on a classroom of children but he is 
interested in saying something about all similar children. Therefore, 
the researcher is quite concerned about using a representative sample. 

The evaluator , on the other hand, is interested primarily in his group 
of students. He will collect data on all the children in his program 
and have no concern about adequacy of sample. If there are only ten 
blind children in his districts special program, changes in the behavior 
of these ten will be sufficient to help him evaluate his program. Other 
districts will just have to do their own evaluation of their own programs, 
although they may use his results in picking techniques which are likely 
to succeed. 

Second, we are usually concerned with an existing complex program 
in which we cannot control all extraneous variables. As researchers , we 
might want to have uniform levels of teaching experience and training 
of our staff, narrow IQ ranges in our classes, and a stable population 
for the length of the study. However, as evaluators , we are satisfied 
to know how effective a program has been with these teachers (including 
Mrs. Smith, who quit in mid-year), in this district (where a school 
board member challenged one of our textbooks as too violent), and 
with these youngsters (including those who really shouldn’t have been 
in the program). Rather than control all of these variables, we must 
aim for a more detailed description of them. 

Although theory-based research and evaluation programs usually 
attempt to state specific hypotheses in advance, both must be alert to 
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the incidental or unexpected results. These frequently occur and may 
be more important than the expected results. A particular piece of 
programed instruction is found to be no better than a teacher but takes 



half the time. A certain perceptual training program improves a student’s 



self-concept. Such are the unexpected twists which add spice to 
educational and scientific endeavors and keep observers of child behavior 
on their toes. 

Special Aspects 

Special educators face all of the evaluation problems common to their 
fellow teachers and can use almost all of the general techniques. They 
may be faced, however, with groups of children who show a great many 
differences amongst themselves. They may be using techniques of in- 
dividualized instruction, or dealing with a very few children of a 
particular handicap. For these reasons, evaluation of programs in 
special education is less likely to use group data or sampling techniques, 
and Is more likely to use detailed description and observation of 
individuals or small groups . This focus frequently necessitates the 
use of descriptive rather than inferential statistics, with an 
emphasis on the unique or ’’special” elements of the situation or an 
individual. 

Evaluation of programs with these ’’special aspects” may seem a 
difficult task. Because of our* emphasis on objectives as guidelines 
for action and on meaningful observation and description, evaluation is 
basically a matter of organization. As we deal with the organizational 
process of over-all evaluation development, we must define what and 



when to evaluate. 
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Chapter II 

WHAT AND WHEN TO EVALUATE 

The underlying principle in this resource guide is the focus upon 
the whole. One must take into account the totality of factors req uir ed 
to reach a certain objective or goal. Furthermore, it is the dynamic 
interrelations among these factors which finally produce the end 
product. As important as it is to keep in mind and be sensitive to the 
whole , it is obvious that there are parts which, in turn, have specifics. 

Educational projects, large or small, will contain three distinct 
but interrelated stages characterized by these specific factors : 

"Where are we? What do we have to work with? What do we want to 
change?" 

Inputs , prior conditions , or antecedents . (OBJECTIVES) 

"How do we get there from here? How do we know we’re moving?" 

Processes , interactions , or transactions ,, 

"Where are we going? What happened? What skills were changed?" 

Outputs , outcomes , or products . ( IMPACT ) 

The General Process 

For example, let us consider the teacher who is preparing to 
teach safety skills in driving to mentally retarded teenagers , It is 
evident that the desired end product (the objective) is the mastering 
of subject matter by the students. He will convert these objectives into 
demonstrations of skills, verbal tests, or an application of these skills 
in simulated situations. The outcomes, products, or outputs will thus 




be derived. 
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Before the teacher begins to teach, he will want some idea of what 
each student knows . He may have recorded information on the prior know- 
ledge of the students or he may give a pre-test. The teacher will be 
aware of some of his own strengths and weaknesses in the subject. He 
will investigate the school’s equipment and materials which are available 
for teaching this subject. What films and teaching aids are available? 

Does the school have a driving simulator? Do the local police or State 
Highway Department have demonstrations of driver safety? All of these 
factors represent prior conditions , antecedents , or input conditions and 
factors to which attention must be given before the instruction is 
carried out. 

After the teacher considers and identifies the above factors and 
perhaps others, he then proceeds to plan and execute the learning activities. 
At this point, he considers how the students will be expected to interact 
with the teacher and with the various materials which he will utilize. 

The physical arrangements and setting will be recognized as a part of 
the procedure. Group and/or individual work will be planned and under- 
taken. The involvement of other teachers, supervisors and administrators 
should be considered. All of these processes , interactions or transactions 
describe what will take place during this stage or phase. 

The teacher, having thought these various aspects of how, when, and 
where these situations and interrelations will occur, will be prepared 
to gather data and information so that they will know how well the 
students did as a result of their planning. As soon as the instruction 
is undertaken, data will be generated which will provide feedback and 




make modifications in the present plans possible or make it possible 
to help another teacher reach essentially the same objective. 

Finally , even though the teacher has certain final objectives in 
mind, he will need to develop procedures for gathering information and 
data as to how well these objectives were met. 

Meager has identified essentially the same points in the first 

paragraph of Chapter I of his book entitled Preparing Instructional 
Objectives : 



Once an instructor decides he will teach his 
students something, several kinds of activity 
are necessary on his part if he is to succeed. 

He must first decide upon the goals (output) 
he intends to reach ax the end of his course 
or program. He must then select procedures, 
content and methods (inputs) that are relevant 
to the objectives; cause the student to interact 
(processes) with appropriate subject matter in 
accordance with principles of learning; and, 
finally , measure or evaluate the student f s 
performance according to the objectives or goals 
originally selected. 

The same plan, as identified above, should be applied to the 
evaluation of a special education project. After initial decisions are 
made and the general context in which the new program will operate is 
determined, the objectives need to be identified. 



The manner of dividing the total process of evaluation into 
manageable parts or stages would likely begin with what teachers also 
would undertake first, namely, to identify the objectives o • outputs 
they were attempting to achieve. 

Outcomes , Products , or Outputs 

The end results or the impact of the study should be identified. 
These will include such outcomes, products or outputs as those pertaining 
to the students, the teacher, the instructional material or the school 
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in general. In some cases evaluation of immediate results will be 
made, in other instances long range or permanent changes will be judged. 
Such areas as the following might be included when considering our 
safety project for mentally retarded pupils. 

I . Pupil 

a. Verbal ability 

b . Interests 

c . Motor skills 

d. Use of and/or preparation of materials 

e. Self-motivation and direction 

II. Home 

a. Changes in parental attitudes 

b. Impact on subjects 

c. Impact on siblings 

d. Changes in activities or habits of subject 

III. School 

a. Changes in attitudes of teachers and administrators 

b. Use of special facilities, i.e. , driving simulations 

c. Staff mobility 

d. Future plans and developments 

e. Teacher diaries or counts 

f . Effects on non-participating students and teachers 

g. Changes in financial support 



Prior Conditions, Antecedents or Inputs 

An attempt should be made to identify and gather all kinds of useful 
information before the project gets underway. This information might 
include data about the pupils, the parents, the teacher and the setting. 

After the desired outputs or end products are determined it seems 
natural to consider what kinds of inputs are necessary to achieve these 
products. An understanding of prior conditions may establish bench marks 
and thus provide clear evidence of change. The following are suggestive 



of what needs to be considered after objectives and outputs are established 



but before the project gets underway: 

I. The Pupil 

a. Sex 

b. Age 

c. Siblings 

d. Visual tests 

e . Hearing tests 

f. Motor ability 

g. Interest inventories 

h . Emotional stability 

i. Relations with peer group 

II. The Parents 



a. Marital status 

b. Socio-economic level 

c. Natural origin 

d. Age 

e. Educational level 

f. Place of residence 



III. The Teacher 



a . Educat ional background 

b. Evidence of recent continuing education 

c. Age 

d. Sex 

e. Previous employment record 

f. Relations to colleagues 

g. History of adoption of new techniques or procedures 



IV The Setting 

a. Physical arrangement of classroom 

b. Availability of audiovisual equipment and material 

c. Type of material being used, copyright date, etc . 

d. Single or multiple text and materials 

e. Light control and acoustics 

f. Equipment for individual, small group or large group use 

g. Preparation of special materials or equipment 



Processes, Interactions or Transactions 



One of the major differences with traditional research procedures 



and the one recommended in this guide is the recognition given to the 
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interaction among pupils, teachers, administrators, instructional 
materials, instructional patterns, ancillary personnel, specialists, 
media and any other process while the project is underway. The 
interactions of the various forces which need study and assessment 
as well as what comes out of or goes into the project is considered 
imperative by the co-authors of this guide . Some of the areas to be 
considered in advance for investigation before the project begins 
are as follows: 

I. Teaching activities 

II . Administrative activities 

III . Student activities and responses 

IV. Activities of specialists 

V . Use and flow of instructional materials 

VI . Classroom organization and interaction 
The impact of this project could be measured by the number of 
teenagers obtaining driver licenses, the change in attitudes about 
retarded drivers and/or increased local support for additional 
programs of this kind. 

Conclusion 

A function of over-all planning is to provide a continuous flow 
of information. These data may require changes in procedures. When 
unexpected events occur, the monitoring process should be sufficiently 
sensitive to suggest new objectives and thus new data gathering. There 
fore, the various initial aspects of the project should be subject to 
change depending on new, modified or selected objectives, outputs 5 
processes or inputs. It is a dynamic and changing situation which we 
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hope to assess, not a static or isolated activity. The over-all 
model is shown in Figure I. 

A risk in breaking down a complicated whole into its parts is 
that the tree may blind one to the forest. It is the interrelations 
of the parts which produce the end product or output. However, it 
must be kept in mind that one is dealing with a system as discussed 
in the opening chapter. Thus, the evaluator must keep a sharp and 
focused view on the individual "bits and pieces" but with frequent 
scanning and judging of the changing, over-all picture. Though the 
evaluator must be aware of the "parts" of the project, he r.ust keep 
in mind that his overall objectives should furnish him with guidelines 
for action. Thus, the development of objectives is of great signifi- 
cance in giving direction to the evaluation process and suggestions are 
made regarding such activities in the next chapter. 

"If telling were the some thing as teaching , we'd all he sc smart 
we could hardly stand it, ” 

Robert Mager 
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Chapter III 

OBJECTIVES - GUIDELINES FOR ACTION 



Motto : "The secret o£ success is clarity 0 / purpose. ". . . 

Benjamin Disraeli 

The first step in the evaluative process is accomplished when we 
determine the scope and direction of our educational plans. Where do 
we want to go? What do we want to accomplish? What purposes do we 
pursue? What goals do we have in mind? Questions like these lead to 
answers which form the basis for a statement of educational objectives. 

We could say that objectives are outgrowths of our thinking about the 
nature and purpose of our intended actions . Thus, they become guide- 
lines for our programs and targets at which we aim our efforts. These 
guidelines must be properly evaluated as the program develops in 
order to help us achieve that " clarity of purpose ! t which Disraeli 
called the secret of success. 

In order to show (1) how to establish objectives and (2) how. to 
develop these objectives in (a) behavioral terms and, (b) in exploratory 
terms, and also to show how the evaluative process is involved in each 
step, let us consider the following practical example by which a set of 
objectives for a program emerged. 

Establishing the Initial Objective 

The first grade teachers in a school district became concerned 
about a large number of children with speech difficulties, which undoubtedly 





originated in pre-school years. After discussions with administrators 
and board members, the decision was made to attempt a change in this 
situation. A group of their colleagues, a speech therapist and an 
administrator checked on published experiences of others (’’They reviewed 
the literature.”); they assessed the available resources in the district 
and the potential aid from other funds; and on the basis of these 
steps two objectives were developed for a pre-school program with speech 
handicapped children. 

1. They decided that it was necessary to locate these children 

and to assess the nature of their defects. Their objective 
read: "In this program we shall screen children with speech 

difficulties in kindergarten and nursery schools and diagnose 
the nature of their handicaps.” 

2. Since a diagnosis without therapy would not produce desired 

improvements in the speech condition of these children, the 
group proposed that a systematic training program be initiated 
in which parents, kindergarten and nursery school teachers 
would be assisted by a speech therapist. Their second 
objective read: "A training program will be initiated for 

all speech ahndicapped children who have been diagnosed 

in the pre-school screening procedures. Parents and teachers 
will be assisted by a speech therapist in this training." 

Developing Behavioral Objectives 



As the planning continued a university consultant suggested that 
the two initial objectives should be expanded and refined. He emphasized 
that, whenever possible, these objectives should be stated in "behavioral 
terms." His suggestion reflected a recent trend in evaluation procedures. 
When descriptive behavioral terms are spelled out carefully, progress 
and improvements in any dimension of learning or behavior can be measured 
or assessed against the specific statement of the objective. Following 
the consultant’s advice, the project planning group expanded their two 
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general objectives into several descriptive behavioral statements. 

They wrote: 

1. Screening of all pre-school children will lead to early 

diagnosis of: 

a. children with function articulatory defects. 

b. children with delayed speech patterns. 

c. children with stuttering symptoms. 

d. children with combinations of the above conditions. 

2. Gearing the proposed training program to the four listed 

categories of diagnosed children, the planning group arrived 

at four expanded objectives: 

a. the training program will reduce articulatory defects 
by 50% after one year of systematic treatment. 

b. the training program will increase interest in oral 
expression in all cases of delayed speech by measurable 
amounts. 

c. the training program will initiate rhythmic exercises 
for all stuttering children and re. uce their level of 
anxiety. 

d. the training program will develop sets of combined 
exercises for all children witn multiple speech-handicaps 
in order to reduce articulatory defects, increase interest 
in vocal expression and produce higher levels of self- 
confidence . 

Further refinement of these objectives was suggested as the program 
evolved. In regard to objective 2a, several behs^oral sub-objectives 
were finally formulated: 

2al all children who substitute a th-sound for an s- sound 

will show improvement in 30-50% of examples on a pre-test/ 
post -test audio-tape measure. 

2a2 all children who omit initial sounds will show similar 
improvements on pre-test /post -test audio-tape measure. 

2a 3 all children who distort endings of words will show 

improvement in speaking of correct endings at a 15-30% 
measured rate on a pre-test/post-test audio tape. 

this point in the process the planning group concluded that: 
Objectives are statements of specific behaviors > which we 
intend to achieve through an education program. 

Plans for training and education should thus be designed for the 
achievements of these behaviors. This postulate imposes a limiting 
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dimension on the execution of a plan: The specifically stated be- 

havioral objectives guide the selection of activities, methods and 
media and prevent us from dispersing energies and efforts on 
peripheral outcomes , 

Developing Exploratory Objectives 

The planning group also suggested that: 

Objectives should contain "exploratory" -possibilities . 

When one conceives of evaluation as a continuous and self-correcting 
system in which all aspects of a proposed program are under constant 
observation by those who conduct it, then objectives also become subject 
to necessary changes when: 

1. we recognize that other favorable objectives could emerge 
in the evaluation process. 

2. we note circumstances which prevent us from achieving the 
stated behaviors. (Evaluation can accept failure, provided 
you change your intentions ! ) 

Two examples will illustrate these conditions: 

A teacher of trainable retarded children had written two objectives 
for a program of improvement of eating habits for some children in her 
group. 

1. The number of undesirable interferences with the eating of 
other children will be reduced in half at the end of a 
six -week training program. 

2. The children will learn 15 new names of foods and will be 
able to name these foods correctly at a 90% level when 
presentation of these foods is made at the end of the training 
period. 

During the third week of her program the teacher notices that in addition 
to the desired reduction, several children began to pay ostensibly more 
attention to their own eating habits. This observation prompted the 
teacher to add subsequent objectives to her program which she stated 



in these terms : 
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1* Fewer spilling incidents 

2. Better manipulation of eating tools 

3. Reduced need for supervision of children at mealtime 

In regard to her second objective the teacher noticed early that she 
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names. Accordingly, she reduced the criterion level of her second ob 



jective from 15 to 10 new names. 

Objectives: Realistic and Relevant 

This last example illustrates two additional dimensions which the 
writer of objectives must keep in mind: 



1. Objectives must be realistic and 

2. Objectives must be relevant. 

Both of these requirements are of great importance to anyone who plans 
to design an educational program. In order to estimate how realistic 
a given set of objectives are , the director of a planned program might 



ask: 



Do I grant enough time for the accomplishment of my objectives? 

Are my resources (personnel, materials, funds) sufficient for the 
execution of my planned program? 

Do experiences in other programs suggest a good probability of 
success for the achievement of my objectives? 

One could say: If you shoot for the moon, make sure you have a 

big enough gun and aim it right ! Or , to relate this statement 
to the realm of educational evaluation: Whenever educational 

objectives are to be met successfully they must be in line 
with the reality of circumstances in which they occur. 

Last, but not least, when assessing the relevance of educational 
objectives, a director of a program is well advised to weigh carefully 
how his proposed special program fits into the total efforts which a 
group, a school, a district, or a state plan to expend. Additional 
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consideration should be rendered to the long-range effects of a program. 
It is more sensible to plan a sequence of short-term objectives which 
fit into the framework of general objectives than to initiate incoherent 
cind. noH“2.n*tGg]?3.*t9ci activities. 

As can be seen in Figure 2, short-term objectives, which all 
contribute to a long-term objective, are "parts" of the "whole." Again 
it should be stressed that objectives are outgrowths of our thinking 
about the nature and purpose of our intended actions . 

In order to determine if the objectives have been reached it is 
necessary to generate data which are related to the objectives. It 
does no good to identify and collect information on how well a deaf 
child learns to read lips if our primary objective is to enable the 
child to relate more positively to his peer group. 



Figure 2 



SHORT AND LONG TERM OBJECTIVES 



SHORT 

TERM 

OBJECTIVES 




LONG TERM OBJECTIVE 



IMPROVE READING LEVEL OF EMR - GROUP 



Develop 
Individualized 
Program 



Create S 
Sustain 
Parents * 



Help 



Test 
Frequently 



Change 

Your 

Approaches 
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Chapter IV 
DATA COLLECTION 

The single most important question which confronts any teacher 
or project director, therefore, is as follows: How do we know whether 

our plan has succeeded? What is the evidence by which we support our 
feelings that a program helped children? Can we be sure that we 
accomplished our objectives? 

The process by which we collect evidence or the medium of proof 
is called data collection. It is a process in which all men are 
constantly engaged as they evaluate their actions or the actions of 
others in daily living. But this everyday data collection is casual, 
often not planned and rarely complete. However, data collection for 
the evaluation of any project is a systematic, orderly and well planned 
process. 

Objectives Determine Methods 

From the discussion thus far, one necessary condition emerges: 

The choice of objectives determines the method of data collection. We 
must look for evidence which will furnish us with seme proof about the 
degree of accomplishments of our objectives. Thus, we should collect 
only such data as are sensibly related to our objectives. To be 
specific: If my objective is the improvement of physical development , 

my data should give evidence on the dimensions in which this development 
occurred. In such a project it would be senseless to collect information 
on mental growth as measured by an intelligence test. If, on the other 
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hand, my objective is an increase of children’s abilities to discriminate* 
to generalize and deduce. Then an intelligence test covering these 
areas of mental progress is entirely justified. 

There are simDle methods of data collection which everv teacher 
can perform. However, one misconception should be dispensed with before 
we begin. All too often teachers shy away from data - collection because 
they confuse this process with the concept of statistics. They forget 
■that statistics are only a mathematical tool to organize data collection. 
Sophisticated statistical treatment is not the sole measure of a good 
evaluation. Its techniques, however, can be most helpful when we organize 
our fota collections. A teacher who does not know the techniques of chi- 
square, or analysis of variance can still produce a respectable evaluation. 
But he must relate the collection of data to his objectives and he 
must use care and prudence in his observations and the recording of 
his findings. Most methods for the collection of data are simple as the 
following examples will show. 

Example I 

Objective: To improve personal grooming and care for clothing in a 

group of trainable, retarded, adolescent girls. 

The teacher for this project chose two methods of data collection: (1) 

photography and (2) the judgment of mothers and of training aides at 
the center. 

Method: The first method consisted of taking three different photographs: 

one at the beginning, one in the middle and one at the end of 
the training program. The pictures were compared by independent 
judges (her fellow teachers in the building). Hair grooming 
and outward appearance were the criteria which had been selected 
for improvement. Care for clothing was judged by a simple checklist 

ERiC 
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which is shown below. These checklists were filled out by 
the mothers and the aides for a period of six weeks and the 
results were tabulated: 



Name : Mary Jane 


Monday 


Tuesday 


Wednesday 


Thursday 


Friday 




C S 


C 


s 


C S 


C S 


C 


S Totals 


Morning 


X 


X 




X 


X 


X 


5 0 


Lunch 


X 




X 


X 


X 


X 


3 2 


Afternoon 


X 




X 


X 


X 


X 


1 4 




C — Clean 






S— Soiled 









The results were compared with a control group which did not receive 
training in these two aspects of personal care and the teacher was able 
to demonstrate that her group had made steady progress ir. the achievement 
of the stated objectives. 

Method: Frequency counts of observed behaviors, judgment of photographs 

on specified criteria and comparison of two groups with different 
treatment procedures. 

Example II 

Objective: To correct misarticulation of V, th, and all sibilants in a 

group of diagnosed children with such handicaps. 

After pre-testing, a chart was established on which the progress of each 
child was measured. The chart listed the following specific objectives: 

1. Articulation of V 

2. Auditory discrimination of n th” 

3. Ability to use ”th” in words and speech 

Ten trials with selected words containing V and th-sounds were conducted 
for eighteen successive days. Individual therapy preceded each evaluation. 
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Number of 10 

Successes 9 

out of 10 8 

Trials 7 




y y-^-X— X —X — X“*X X — *x 

X-A .0-0-0 



4 

3 

2 

1 




O' 



xO-O-O" 




Day number 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 
In addition, the child’s performance was taped on a tape recorder and the 
recordings were compared with tne rerults on the charts . 

Methods used: Graphic illustration of individual performance record and 

taped verification of improvementr. 

Example III 

Objective: To reduce undesirable behaviors in a group of emotionally 



This teacher combined her own frequency counts of the children’s un- 
desirable behaviors with checklist of desirable behaviors the children 
kept for themselves. In addition, we used a Sony video tape recorder 
to film three specified situations at frequent intervals. These situations 
were viewed and discussed by the children and a group chart was then 
developed by the children. 

Example of Group Chart: 

Activity: Entering the Room YES NO 

• Mont children went to their seats 

Most children began to study their assignments 
Most children paid attention to teacher’s words 



disturbed children. 
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Method: Cumulative frequency counts of undesirable behavior on 

individual checklists (teacher observation). Cumulative 
frequency counts of desirable behavior on individual 
checklists (child self-observation). Filming of special 
scenes, discussion of video tapes, and group adjudgment of 
observed behaviors. 

In this project frequency counts of observed behaviors were collected 
over a period of a year. Significant trends of improvement in the 
direction of stated objectives were observed. The teacher also noted 
that the group charts kept by the children had a significant reinforce- 
ment effect on the children’s behavior. 

Example IV 

Objective: To acquaint mentally retarded children with industries in 

the community and to measure increased knowledge with 
appropriate instruments. 

Several field trips were arranged to carefully selected industries 
in the community. The teacher devised tests of twenty items from each 
field trip. Results of the tests taken after the field tests were com- 
pared to pre-test results. Test One consisted of the following items: 

1) Five taped items in which sounds in an auto body shop 
were portrayed. 

2) Five pic tores (slides) taken in the shop visited. The 
children had to identify the situation of work portrayed. 

3) Five drawings of tools used in the shops. The children 
had to identify the tools by name. 

4) \ive incomplete sentences which were read to the children 
and called for verbal completion, i.e. , Bent fenders are 
smoothed with a ( hammer ) , or Vfhen the fender is finished the 
sprays it with ( paint ) . 
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Method: Comparison of successful item completion on a teacher-made 

test designed to measure differences in pre- and post-test 
knowledge of local industries by mentally retarded children. 



Example V 

Objective: To expand vocabulary of children with severe speech handicaps 

Ten activities were selected in which a puppet theatre was used to 
exemplify the use of 100 target words. No children with hearing defects 
were included in the sample. The ten situations portrayed scenes in: 



a grocery store 
a fire station 



a kitchen 



a bakery 



a zoo 



a luandry 



a post office 
a gasoline station 



a barbershop 
a clothing store 
For each situation ten target words were selected for which the 
teacher developed a simple picture-identification test. Pre- and post- 
test results were compared and the t-test for signifi.cance of differences 
was applied. An example of one situation is portrayed below. (Each target 
word was presented at ten different occasions in order to measure lasting effects), 
Example : Grocery Store 

Target Word : BUTTER 



CORRECT IDENTIFICATION 



Date 



Yes 



No 



2-5 x 

2-8 x 

2-11 x 

2-14 x 

2-17 x 

2-20 x 

2-23 x 

2-26 x 

2-29 x 
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Method: Cumulative frequency of correct identification of selected 

target words. Pre- and post-test comparison. T-test for 
statistic significance of recorded differences. 

In addition to these examples , other appropriate methods of data 
collection can be employed. Certainly the use of standardized tests 
should be recommended. However, we want to emphasize that very few 
good and reliable standardized tests for the measurement of handicapped 
children are presently available. Questionnaires and opinionaires can 
also be most helpful in the collection of significant data . If progress 
is to be judged by parents or community groups, these latter instruments 
usually provide for economic, and within limits, reliable sources of 
information. 

Last but not least, a list of data collection instruments must 
recognize the contributions which well organized diaries or case 
studies can render. When longitudinal effects of a treatment are to 
be assessed, continuous record keeping with these devices can provide 
exceptionally valid data for the evaluation of a project. A word of 
caution should be expressed: It makes little sense to write books on 

behaviors which are not relevant to objectives. The collected data must 
be meaningful for the specific project. It must also be assembled in 
a logical manner so that the significant aspects of the project may be 
assessed and communicated. 

Because the amount of money which is available will determine the 
size and nature of the project being planned, attention is given to 
it next . Since the amount of financial resources is seldom if ever 
adequate, the use of limited resources to achieve the objectives should 
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be considered by those who evaluate proposals for funding as well as 
those who direct projects which are funded. For these reasons the budget 
is considered next as it relates to evaluation. 
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Chapter V 

BUDGET PLANNING AND EVALUATION 

All planning and programming of educational activities must be 
related eventually to the systematic management of financial affairs 
which we commonly describe with the terms budget and budgeting. 

In the context of this guide we are primarily concerned with two 

questions : 

1) How does budget planning relate to the total process of 
evaluation? 

2) How do evaluators look at the budget section as part of 
a total educational program? 

The wording of the questions suggests desired limits in this 
section of the guide: It is not our intention to present a detailed 

?, How to do” prescription on budget -making. This service is adequately 
performed by the manuals and guidelines of federal programs, and careful 
reading and study will enable the writers of proposals to compose the 
documents as specified. Here we propose to state only the relationship 
of budget to the process of evaluation. 

Budget Planning and the Total Process of Evaluation 

The writer of a proposal or the planner of an educational program 
may well be advised to raise and answer a set of questions which can 
illustrate and clarify the relationship of dollars (budget) to the 

input of the project. For example: 

1) How much personnel service will be needed (input factors) 
in order to achieve a workable student -teacher ratio? 

If eighty percent of projected funds are put into this 
program effort, will there still be sufficient funds for 
the purchase of necessary equipment? 



2) 
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3) Can favorable cost -result ratio be achieved if most of the 
available resources in personnel-equipment-expertise , etc , 
is allocated? 

While this set of questions illustrates possible effects of budget 
on planning, other questions might point to xhe necessity of alternative 

planning in the budgeting process. 

1) What will happen if the project can only be funded on an 
eighty percent level? 

2) What local resources can be mobilized in order to augment 
budget and continue with programs after federal support 
will cease? 

3) Has the planning considered all available services of 
libraries, IMS’s or other local, state or federal agencies 
to reduce budget costs? 

4) Has information from similar projects been used in order 
to obtain maximum return from dollar investment on the 
basis of previous experiences? 

In summary, all budget planning must contain a careful assessment 
of the probably effect dollar allocations will have on existing conditions 
(input). It must further consider what interaction of factors will be 
accomplished (process) and last but not least, it must estimate as 
accurately as possible what effect dollar inputs will produce in terms 
of accountable results (outputs). The intimate relationship of these 
considerations to the process of evaluation is self-evident: Budget 

planning represents projected financial record of program activites. It 
is the very purpose of evaluation to assess the outcome of these activities 
against postulated goal** and hence by inference to measure the effect 
which dollars (as a major factor of input) produce in the program. 
Evaluators and the Program Budget 

Nobody can precisely predict how a given panel might specifically 
©^aluate a proposed budget. However, the overriding concerns for budget 
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evaluation can be postulated with the three terms: Efficiency - economy - 

effectiveness. 

A budget contributes to efficiency if the projected expenses facili- 
tate process interaction i.e.when monies are available for the right 
effect at the right time and in sufficient quantity to bring about the 
desired changes. 

A budget contributes to the economy of the project by assuring 
maximum effeGt with minimum outlay and by having clearly established 
priority allocations. Economy is most easily assessed vjhen long- 
range program planning shows the projected outlay for a number of 
years, with each year logically and financially related to the total 
program and the total expense. 

A budget contributes to effectiveness if available sums are 
carefully assessed at points of decision. 

In summary, evaluation and budgeting can interact to produce more 
effective project development. Budget planning relates directly to the 
output input and interaction phases of the total process of evaluation. 

A budget which contributes to efficiency, economy and effectiveness can, 
in turn, be meaningfully assessed by project evaluators. 
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Chapter VI 

KELP AND HOW TO USE IT 

In most projects those who conceive and develop them will be 
able to develop the appropriate evaluation procedures. It is hoped that 
this guide will be helpful. In addition* the references found in the 
Appendix should be useful in providing further information on a variety 

of aspects of evaluation. 

For those who feel the need to use consultants there are two 
main ways in which such consultants or experts can be utilized to 
help strengthen your programs: at the individual project level and 

at the state department level. The last section will deal wii.ii _esourwes 
that are available and how they might be of help to your individual 
projects. 

Project Consultants 

A project consultant should be someone in the local area who knows 
something about the type of child or treatment approach proposed. You 
should try to get him involved at the earliest possible stages in your 
project so that he can make suggestions and give constructive criticisms 
on basic ideas. Good consultants, like other specialized personnel, 
should be compensated in accordance with their contributions and can be 
included as a item in the budget. The following is a list of duties 

which might be expected from a consultant: 

1. Review previous fin dings . A consultant should be able to 
provide you with information concerning similar projects, 
what the problems were and make suggestions about what needs 

to be done. 
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2 . Help plan objectives . The chances are you will already have 
a set of objectives. A good behaviorally oriented consultant 
can help you pinpoint your objectives and spell them out 
clearly, which in turn should make your teaching and evaluation 
plans easier. 

3* Provide resources- This person should be able to provide 
up-to-date information concerning the best instructional 
materials available and point out evaluation and reading 
materials that will be helpful. 

4. Help plan methodology . If the consultant is chosen wisely, 

he should have information concerning the teaching or treatment 
approach to be used. He can help with the many subtleties of 
programming and procedure to achieve the objectives. 

5. Help with design . Although Title VI Projects are not intended 
to be rigorous research efforts, it is only fitting that those 
who are spending money on the project will be interested in 
the effectiveness of the program. The expert should be able 
to indicate appropriate methods of assessment that will allow 
the making of clear-cut statements concerning the effect of 
the program on the intended subjects. 

6. Assist in recording of findings . The consultant should be able 
to assist in the systematic recording and assessment of findings 
while the project is being conducted about the progress of 
students toward the specified objectives. 

7. Testing. It is possible that a specialized instrument has 
been chosen to measure the effectiveness of the program. (This 
is certainly not a requirement but if the project seems to demand 
it the consultant may be able to advise about instruments which 
do not require such sophisticated skills. The consultant could 
serve himself as an examiner where outside testers are required) 

8. Analyze data . It may be desired to have the consultant organize 
and analyze "the data that have been collected and put it in some 
general form that can be easily reported. This is especially 
true if the specialist was responsible for the testing. 

9. Help write the final report . Depending upon the amount of 
involvement that this person has had with the project, he 
might be quite helpful in assisting in writing an orderly 
final report. 

The preceding list of suggestions is just that — suggestions. One 
may use all, or none, of the services indicated. If one has adequate 
staff to provide all of the functions mentioned above a consultant need 
not be employed. If, however, a staff is harried and overworked already, 
a consultant may be exactly what is needed. 
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State Consultants 



A distinction has been made between state consultants and the 
project consultants. The basic purpose of a state consultant is to 
provide those state departments of education, who desire such assistance, 
with a third-party evaluator concerned with the procedures and ev al uation 
of all programs. A project consultant usually works with only a single 
project. These two types of consultants should probably be different 
persons. If the state consultants also provide detailed service to 
individual projects, then the problem of conflict of interest arises. 

The following is a list of suggestions of how third-party evaluators, 
acting as state consultants, should be able to provide services to 
state departments: 

1. Advisors to selection committee . When the state departments 
gather their forces together to review Title VI proposals, it 
would seem reasonable to have the state consultants on board 
at that time to help in the selection process. 

2. Meet with directors . The directors of approved projects should 
meet at a central location and individually spend at least an 
hour with the state consultants. At this time the projects 
are reviewed in detail and the final objectives and evaluation 
procedures can be pinpointed and agreed upon. 

3. Formulate a state evaluation plan. It has been found most 
helpful (Oregon Title VI Impact Study) if the state consultant 
team write a short, follow-up letter to each of the directors. 

It should re-state what was agreed upon in terms of evaluation 
procedures and in what form the data will be reported. This 
will provide simplified guidelines which can be easily referred 
to and will allow both parties to know what is expected. 

*+. Visit projects . State consultants should make at least one 
visit to each of the projects after they have been underway 
for a short time. The purpose of these visits is twofold: 
a) It ^provides an opportunity to take a first-hand look at 
the projects and to insure that communication concerning 
evaluation procedures is clear; and b) It should provide a 
degree of flexibility for the project directors. If some of 
of the evaluation procedures are now inappropriate, new 
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evaluation pi-ocedures that the project can effectively 
accomplish should be mutually developed. These then become 
the new source of data for evaluating the project. Tor 
example, perhaps original objectives were to measure language 
on the ITPA. It is found at the beginning of the project 
that the test is too difficult or that a qualified administraor 
of the test cannot be found. Then, another measure of less 
difficulty like the Peabody Picture Vocabulary Test might have 
to be substituted. 

5. Collect all data . On completion of the project the various 
project directors should send all of their data from the 
evaluation procedures directly to state consultants. It is 
hoped that et this point all project directors can keep their 
data as simple and clean as possible. 

8. Analyze data . If there are any statistical manipulations 
that have to be made, the state consultant could provide 
this service. It is generally hoped that projects would not 
require complex statistical formulas; however, the ability to 
manipulate these should be taken into account when the 
state consultants are selected. If evaluation procedures 
can be kept simple and clear-cut, then the data analysis becomes 
a less difficult job. (Graphs and charts are a big help.) 

7. Write the over-all report . Each individualized project should 
be summarized by the outside evrluators and statements made 
concerning strengths and weaknesses, successes and failures, 
of each of the programs. This final report should be sent to 
the state department for its stamp of approval. 

The basic model presented in the above section is very similar 
to the one used in Oregon on their Title VI Impact Study. (See appendix 
for further information as to how to obtain this report.) There are 
still some questions that should be answered. Per example, what kinds 
of people do you select as consultants? One of the first qualifications 
should be that they know something about education and have been involved 
with children and teachers at a project level. They should be sensitive 
to the qualities of evaluation that ere spelled out elsewhere in this 
document. They must realize that projects are usually not rigorous 
experimental research projects; however, they must keep in mind that 
the effectiveness of the various methodologies used in the individual 



35 



projects are yet to be determined. They should probably not be members 
of the state department staff, but rather brought in from a university, 
college, or research setting that would typically have such a person 
on its staff. 

Resources of Materials and Information 

In the area of special education there are two main resources 
which provide materials and information relating to educational programs 
for handicapped children. The first resource is the fourteen Instructional 
Materials Centers for handicapped children which have been established in 
cooperation with the Division of Research, Bureau of Education for the 
Handicapped, U. S. Office of Education. They collect instructional 
materials and aids such as braille books, test kits, and tapes and 
recording devices. The centers also engage in research and development 
aimed at devising improved teaching materials for the handicapped. In- 
stitutes and workshops are held at the centers to familiarize teachers 
with the use of special educational materials. Handicapped children and 
youth to be served by centers include the mentally retarded, emotionally 
disturbed, crippled, speech- impaired, deaf, and visually handicapped. 

The centers in operation are listed below. Correspondence should 
be sent c/o the ’’Special Education Instructional Materials Center.” 



Center 


Region Served 


Michigan State University 


Michigan 


Room 218 , Erickson Hall 


Indiana 


East Lansing, Mich. 48823 


Ohio 


University of Wisconsin 


Wisconsin 


2570 University Avenue 
Madison, Wis. 53706 


Minnesota 


American Printing House for 
the Blind 

1839 Frankfort Avenue 
Louisville, Ky. 40206 


National 
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Center 

4. Colorado State College 
Greeley, Colo. 80631 


Region Served 
Colorado 
Montana 
Wyoming 
New Mexico 
Utah 


5. Department of Special Education 


Illinois 



Superintendent of Public Instruction 
316 South 2nd Street 
Springfield, 111. 62706 



6. University of Texas 
304 West 15th Street 
Austin, Tex. 78701 


Texas 

Louisiana 

Arkansas 

Oklahoma 


7. University of South Florida 
Tampa, Fla. 33620 


Florida 
Alabama 
Georgia 
Mississippi 
S . Carolina 


8. University of Oregon 
1612 Columbia Street 
Eugene, Oreg. 97403 


Oregon 

Alaska 

Hawaii 

Idaho ; 

Washington 


9. University of Kentucky 
641 S. Limestone Street 
Lexington, Ky. 40506 


Kentucky ; 
Tennessee - 
N. CArolina ; 
West Virginia 


16. University of Southern California 
17 Chester Place 
Los Angeles, Calif. 90007 


California 

Nevada ! 

Arizona 


11. Boston University 
School of Education 
765 Commonwealth Avenue 
Boston, Mass. 02215 


Massachusetts ' 

Connecticut 

New Hampshire 

Maine 

Vermont 

Rhode Island 


12. George Washington University 

Department of Special Education 
820 20th Street, N.W. 
Washington, D. C. 20006 


District of Columbia 

Delaware 

Maryland 

New Jersey 

Pennsylvania 

Virginia 
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13. 



Center 

University of Kansas 
School of Education 
Lawrence, Kans. 66044 



14. 



Region Served 
Kansas 
Iowa 

Missouri 
Nebraska 
N - Dakota 
S. Dakota 

New York State 



New York State Department of 
Education 

Bureau for Physically Handicapped 
Children 

Albany, N. Y. 12201 

The other resource is the Eric Clearinghouse on Exceptional Children. 
The address is 1201 Sixteenth Street, N. W., Washington, D. C. 20036. 

This is a national clearinghouse to organize and disseminate information 
related to education of children and youth who require special services* 
"EriC ExCarpt" currently published as a department of Exceptional 
Children , is reprinted for special distribution on a complimentary basis. 
Individuals in school units will be placed on the mailing list upon 
request. 

In summary, creativity, not project procedure, is the basic ingredient 
in programs. However, the proper use of consultants and resources can 
he the difference between merely solving an immediate problem and pro- 
viding an effective contribution to special education through better 
evaluation and reporting procedures. 

When a project is completed there are the usual final reports to 
be made. In addition to or as a part of these reports, materials should 
be prepared in a variety of formats to inform those interested in the 
project for a variety of reasons. The gathering and producing of 
media should be a part of the over-all plan and should be gathered 
systematically as were the data. The next chapter suggests different 
approaches to reporting and disseminating findings about a project. 
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Chapter VII 

REPORTING AND DISSEMINATION 

The process of reporting and dissemination may be summarized into 
one word: communication. However, this must be effective and available 

communication. Some persons believe that reporting is just bringing 
together statistics and data. Then, it is another’s responsibility 
to interpret and apply what has been learned. No one is in a better 
position to report and disseminate information to other practitioners 
than those directly involved in the new approach. Such persons 
should be in the best position to identify and record the relevant 
information of greatest significance to their colleagues. Reporting 
and disseminating information is meaningful ways is a necessary 
responsibility for anyone undertaking new programs. 

Necessity for Evaluation 

There are several purposes for carrying on studies and evaluating 
and reporting the results. One is to determine if the results are 
favorable enough to warrant continuing the new approach into a similar 
situation. This might be in the same classroom, with the same teacher, 
same equipment and materials but, with a new group of youngsters. Perhaps 
the decision to be made is whether the new idea might be attempted with 
equal success in another neighborhood. 

Another need is to acquaint other schools with the results of 
an experience and to assist them in deciding whether or not to try the 
new approach. The purpose for pilot and exemplary projects is to 
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spread, as widely as possible, ideas which have demonstrated their 
merit and worth. Thus, a clear statement of what happened (when, 
wher. , to whom, and with what effect) is necessary to guide those 
responsible for the new development. 

In either case, sufficient data and information need to be available 
about the initial trial in order to duplicate the experience to ensure 
equal or greater success. 

Purposes Served 

There are several purposes which reporting might serve. First, 
provide sufficient information so that the same individual or other 
persons would be aware of where modification and changes might be 
made in order to improve the project. Thus, the evaluation report 
would serve as a means of feedback for gaining greater effectiveness 
from a similar effort. 

Second ? orient newcomers in order to interest them in pursuing 
the idea further. In such cases, the nature of the report would 
generally be in some audiovisual format acquainting uninformed 
individuals or groups with the general purpose, mode of operation and 
evidence of success of the current project. The function of the 
reporting in this instance would be to stimulate interest and provide 
motivation to examine the idea in greater detail for possible application 
to another situation. 

A third purpose of reporting would be to give sufficient detail 
to an outsider so that he could decide if it would work in his situation. 
For this purpose, specific information about the students and what 
was done to them should be included. 
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Reporting procedures should be of such a nature, therefore, that 
these three purposes and perhaps others could be served. The same material 
might suffice for all of these purposes if properly conceived and 
developed . 

Method of Reporting and Disseminating 

Often reporting and disseminating have been thought of as merely 
’'paperwork." This is obviously untrue when put in the context of 
communication. There is a variety of forms in which reports might be 
developed which could assist greatly in promoting new practices. 

1. Visitation . Educators seem to be particularly impressed by 
visitation to actual projects. To "go and see" is an expensive 
and time consuming activity but has been found to be particularly 
useful in promoting and initiating new projects. If a project 

is one to which people are likely to come to visit, their 
needs bhould be anticipated. Administrators wish to visit 
with administrators and teachers wish to observe and talk with 
teachers. Visitors may also wish to observe and interrogate 
students and their parents. Unless these matters are considered 
carefully, visitation can be chaotic and unprofitable. Therefore, 
one way of disseminating information is to arrange for visitors 
to be systematically involved in observation and discussion 
with key people. They thus gain the necessary insights to make 
valid judgments 

2. Reports . Written reports are a necessary adjunct to other kinds 
of information. It is probably true that the usual verbal and 
statistical reports have not changed very much practice. It 
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would be useful if some standard format could be developed such as that 
used in reporting Title VI programs in the State of Oregon. (See 
Appendix reference to IMPACT of the Title VI Programs in the State of 
Oregon prepared under the direction of the State Department of Education 
by the Teaching Research Division.) 

The outline follows: 

Project Title 

Type 

Member 

Funding Allotted 

Background and Rationale 

Ob j ectives 

Methodology 

Evaluation 

Results 

Written repots can be made much more interesting and worthwhile 
if graphs, cartoons and illustrations are included in the text. Meaningless 
tables and other data can often be transformed into easily interpreted 
and useful information. 

In addition to visuals added to the text, charts, visual displays 
or transparencies might aid in the understanding and interpretation of 
the material. 

3. Use of Media in Reports . It has been found that carefully 

prepared audiovisual displays greatly assist in the portrayal 
of newer instructional programs. An attempt to ’’show it like 
it is” cnaracterizes mediated presentations. These are particu- 
larly useful for introducing and orienting individuals tc new 




ideas as discussed above. Among the various mediated 
approaches being utilized currently are the following: 

a. Tape-slide kits . These are quite inexpensive and yet 
easy to develop and assemble. New low cost 35mm cameras 
and high speed film make it inexpensive and feasible to 
capture classrooms and events in their natural setting. 

Slides accompanied by an explanatory narrative make a 
simple set-up for display and distribution. 

b. Eight mm film. Many schools are utilizing the 8mm movie 
camera to capture significant aspects of a new program. 

Again new cameras make possible highly flexible and 
meaningfxil motion pictures. Certain manufacturers are 
also advertising the use of a cassette tape recorder x-zhich, 
xfhen synchronized with the 8mm film, makes feasible a 

low cost sound motion picture. 

c. Audio and video tape . Both of these recorders are being 
utilized for evaluation and reporting purposes. By use 
of either the reel to reel or the newer cassette recorder, 

it is possible to capture the emotionally charged expressions 
of the youngsters, the concern or satisfaction of their 
parents, and the trained judgments of the teachers and 
administrators. For some purposes the video tape recorder 
may have an advantage over the audio type, especially where 
it is worthwhile to include movement. With many schools now 
having available portable video tape recorders, this method 
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is a practical manner of disseminating information. 
Difficulties may be encountered xn having sufficient 
light available and in obtaining good audio recordings. 



Conclusion 

Project d ir ectors should anticipate disseminating information xn 
a variety of forms • By including suggested formats in the original 
proposal, information can be systematically captured during the time 
that the activity is in progress. At the conclusion of the project, 
it is carefully organized and interpreted to serve the various 
general and specific audiences. Only by proper reporting and dis- 
semination of information can a project director feel that he is 
fulfilling all of his responsibilities to the project. And only then 
will others in the educational community be aware of modifications, 
interpretations and applications of current projects designed to 
provide the best education possible for our children. 



Appendix A 

REFERENCES ON EVALUATION 



In this section an attempt has been male to be highly selective 
in regard to the kinds of references which are listed. Each citation 
is followed by a brief discussion of the contents of the publication 
to make it of'inost use to those involved in various capacities related 



to evaluation 
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t he New England educational project . 1966. pp. 1-37. 

Prepared to guide local school systems in evaluating ESEA and 
other products, this workbook gives a step-by-step process for 
evaluation. A glossary of terms is included. * 

Educational Product Report, Vol. 2 No. 5, ’’Educational Evaluation: 

Theory and Practice," Educational Product Report, New York, N. Y., 
February, 1969, pp. 1-48. 

This issue is devoted to the various problems concerned with 
evaluation. One chapter deals with a preliminary instrument 
on assessing curriculum materials and another with a product 
study. Three chapters deal with the question of evaluation 
as contrasted with experimental research and scientific 
studies. These three chapters just referred to will be of 
particular interest to those who are concerned about whether 
or not evaluation differs from research. 

Eidell, Terry L. and Klebs, John A., Annotated Bibliography on the 
Evaluation of Educational Programs, November 1968. ERIC Clear- 
inghouse on Educational Administration , University of Oregon, 
Eugene, Oregon, pp. 1-15. 

This publication identifies significant literature found in 
’’Books, Pamphlets, and Papers” and ’’Journal Articles” dealing 
with evaluation. A brief statement is made regarding signifi- 
cant points covered in the item cited. For example, the 
narrative in this section which is followed by an asterisk (*) 
was taken from the publication. This publication should be 
studied by those who wish to go into evaluation in some 
detail or depth. 
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Grobman, Hulda , Evaluation Activities of Curriculum Projects , Number 
2, AERA Monograph Series on Curriculum Evaluation, Rand McNally 
& Company, Chicago, 1968, pp. 1-136. 

Dr. Hulda Grobman has been very much involved in the development 
of BSCS (new biology) program. Chapters 1, 2, and 4 will be of 
particular interest to those involved in evaluation as described 
in this publication. Chapter 1 considers the concept of evaluation 
(similar to that proposed in this publication) with a final section 
dealing with "Project Evaluation contrasted with other Educational 
Research.” In Chapter 2 on ”What to Evaluate” Dr. Grobman con- 
siders many of the same items presented in Chapter III of this 
publication such as the evaluation of the "process" and the 
"product.” Chapter 4 on "How to Evaluate" is particularly helpful 
in considering the instruments for gathering data. Her chapter 
considers many of the items and techniques considered in this 
publication in Chapter IV entitled "Data Collection." 

A n umb er of actual cases of curriculum evaluation are presented 
so that the reader can make applications of what Dr. Grobman 
discusses . 

Guba, Egon G., The Failure of Educational Evaluation. Educational 
Technology , A Special Report, May, 1969, pp. 29-38. 

A discussion of need for change from traditional methods of 
evaluation to the provision of sensible alternatives. Guba 
discusses "Some Clinical Signs of Failure" in traditional 
evaluation approaches, "The Basic Lacks" along with some 
suggestions for "Where Next?" 

Impact of the Title VI Programs in the State of Oregon . 

Prepared under the direction of the State Department of Education 
by Teaching Research Division, Oregon State System of Higher Educa- 
tion. October, 1968, pp. 1-168. 

A must for anyone interested in evaluating programs for handicapped 
children. After a brief introduction in which among other things, 
a Title VI administrative flow chart for the state level is pre- 
sented, there follow project descriptions. These project descrip- 
tions follow the outline presented in Chapter VII. This publication 
suggests the desirability of establishing a uniform format for 
reporting. This publication is the best single volume available 
to actually include a variety of projects for handicapped children 
which calls for a range of objectives, instruments and methods. 

Lawrence, Teresa M. , Editor s Teaching Exceptional Children , NEA, 1201 
Sixteenth Street, N. W. , Washington, D. C. 

Teaching Exceptional Childr en i s published four times a year and 
is an example of a publication with articles of a practical and 
functional nature which are written to improve practice. Although 
not exclusively devoted to evaluation it is a good example of a 
style of writing and communication intended to assist those involved 
in decision making at classroom levels. 
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O’Keefe, Kathleen Gnifkonski, Methodology for Educational Field Studies , 
Evaluation Center, The Ohio State University, College of Education, 
Columbus, 1968, pp. 1-203. 



Dr. O’Keefe considered ways of gathering valid data and information 

st curriculum projects 
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conducted in actual classroom situations might be classified 
as field studies, her dissertation has much to offer in that 
she considers such matters as expert opinion, as derived from 
the legal profession and how much judgments might be made valid 
when applied to educational projects. When external evaluation 
is called for in a project it usually has elements of tne expert- 
witness idea in law. There are many other useful ideas considered 
in this study which should be very helpful in evaluation. 



Sorenson , G . A new role in education : The Evaluator • UCLA Evaluation 

Comment , 1 (January 1968), pp. 1-4. 

Sees the evaluator in education as a new professional who 
bridges the gap between the university educational laboratory and 
the teaching field, helping teachers arid administrators to define 
their goals, and to appriase their pupils making possible the 
development of instructional programs which will result in 
maximum learning for each student.* 

Stake, R. E. The countenance of educational evaluation. Teacher s 
College Record , 68 (April 1967), pp. 523-540. 

Emphasizes the need for both descriptive and judgmental evaluation 
of educational programs, requiring analysis of three bodies of 
information, distinguished as antecedent, transaction, and out- 
come data. Matrix and flow charts illustrate the evaluation 
process.* 



Stuff iebeam , D. L. A depth study of the evaluation requirement. Theory 
into Practice , 5 (June 1966), pp. 121-133. 

In a special issue devoted to the Elementary and Secondary Education 
Act of 1965, evaluation is presented as a corollary of each of the 
four primary phases of the change process: research, development, 

diffusion, and adoption. Process and decision functions are 
illustrated by a feedback control loop with steps outlined for 
the evaluation of local, state, and federal project operations. 

Nine ways to improve evaluation of educational programs are 
recommended . * 

Stuff Iebeam, D. L. Toward a science of educational evaluation. 
E ducational Technology, 8 (July 30, 1968), pp. 5-12. 

Defines evaluation as the provision of information through formal 
means to nerve as rational basis for making judgments in decision 
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situations. Evaluation methodology includes the collecting, 
organizing, analyzing, and reporting of information, in addition 
to the initial focusing of the evaluation. Four strategies for 
evaluating educational programs (context, input, process, product) 
are distinguished by objective, method, and relation to decision 
making.* 

Tyler, Ralph, Gagne, Robert, and Scriven, Michael. Perspectives of 
Curriculum Evaluation , Number 1, AERA Monograph Series on 
Curriculum Evaluation, Rand McNally S Company, Chicago, 1967, 

pp. 1-102. 

In an introductory chapter Robert E. Stake discusses why achieve- 
ment tests are not appropriate for evaluation of new curricula 
and instruct ional innovation. The chapter by Scriven provides an 
outline of some of the major aspects of any program to which 
attention should be given in evaluation. 




